How metric recording works with diginsight and Opentelemetry

📑 Table of Contents

• Understanding Diginsight Metrics
• What are Diginsight Metrics?
• Tags and Low Cardinality
• Filtering Metrics with IMetricRecordingFilter
  • OptionsBasedMetricRecordingFilter
  • HttpHeadersSpanDurationMetricRecordingFilter
• Enriching Metrics with IMetricRecordingEnricher
  • OptionsBasedMetricRecordingEnricher
• Custom Metric Recorders
• Complete Configuration Example
• References

Understanding Diginsight Metrics

Diginsight provides automatic span duration metrics collection that seamlessly integrates with OpenTelemetry. These metrics measure the execution time of operations (activities/spans) throughout your application.

What are Diginsight Metrics?

Diginsight metrics are automatically collected through MetricRecorder classes, with the primary recorder being SpanDurationMetricRecorder. This recorder:

• Listens to activity lifecycle events using the .NET ActivityListener mechanism
• Automatically records duration when activities complete
• Exports metrics via OpenTelemetry to your monitoring backend (Application Insights, Prometheus, etc.)

Key metric: - diginsight.span_duration: Records the duration in milliseconds of each activity/span with tags like: - span_name: The operation name - status: Activity status (Ok, Error, etc.) - Custom tags from activity attributes

Example:

public async Task<Order> ProcessOrderAsync(int orderId)
{
    // Diginsight automatically creates instrumented activity
    using var activity = ActivitySource.StartMethodActivity(new { orderId });
    
    // Your business logic executes
    var order = await GetOrderFromDatabase(orderId);
    await ValidateInventory(order);
    await ProcessPayment(order);
    
    // when activity stops, metrics are automatically recorded:
    // diginsight.span_duration{span_name="ProcessOrderAsync", status="Ok", orderId="123"} = 250ms
    
    return order;
}

How Metrics Are Sent

Metrics flow through this pipeline:

Activity Creation → ActivityStopped Event → SpanDurationMetricRecorder 
    → OpenTelemetry Metrics Pipeline → Exporters (App Insights, Prometheus, etc.)

Startup Configuration:

// Register Diginsight metrics collection
builder.Services.AddSpanDurationMetricRecorder();

// Configure OpenTelemetry to export metrics
builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics => metrics
        .AddMeter("Diginsight.Diagnostics")  // Listen to Diginsight metrics
        .AddPrometheusExporter()             // Export to Prometheus
        .AddApplicationInsightsExporter());  // Export to Azure

Tags and Low Cardinality

Metrics can include tags (dimensions) that enable powerful filtering and aggregation in monitoring dashboards. However, tags must have low cardinality to avoid telemetry storage explosion.

Why Low Cardinality Matters

High cardinality (many unique values) causes: - ❌ Storage explosion: Each unique tag combination creates a new time series - ❌ Performance degradation: Query times increase exponentially - ❌ Cost increases: Monitoring systems charge based on unique time series

Examples:

Tag Type	Cardinality	Suitable?	Reason
customer_tier (premium, standard, free)	Low ✅	Yes	3 possible values
region (us-east, us-west, eu-west)	Low ✅	Yes	~10 possible values
operation_type (read, write, delete)	Low ✅	Yes	3-5 possible values
customer_id (UUID per customer)	High ❌	No	Millions of values
order_id (UUID per order)	High ❌	No	Unlimited values
request_id (UUID per request)	High ❌	No	Unlimited values

How to Add Tags Properly

Tags are added through IMetricRecordingEnricher implementations. Diginsight provides:

1. OptionsBasedMetricRecordingEnricher: Configures tags via appsettings.json
2. Custom enrichers: Implement IMetricRecordingEnricher for dynamic tagging

Best practices: - ✅ Use categorical values (status, tier, region, environment) - ✅ Use bucketed values (order_value_bucket: small/medium/large instead of exact amounts) - ✅ Keep unique combinations < 1000 per metric - ❌ Avoid user IDs, order IDs, request IDs as tags - ❌ Avoid timestamps, URLs, or freeform text

Filtering Metrics with IMetricRecordingFilter

Not all activities need metrics. Filtering reduces costs and noise by recording only relevant operations.

The IMetricRecordingFilter Interface

public interface IMetricRecordingFilter
{
    bool? ShouldRecord(Activity activity, Instrument instrument);
}

Return values: - true: Force recording - false: Prevent recording - null: Defer to default configuration

OptionsBasedMetricRecordingFilter

Filters activities based on patterns in configuration.

Configuration:

{
  "OptionsBasedMetricRecordingFilter": {
    "ActivityNames": {
      "MyApp.Orders.*": true,           // Record all order operations
      "MyApp.Database.*": true,         // Record database operations
      "Microsoft.AspNetCore.*": false,  // Exclude framework activities
      "System.*": false                 // Exclude system activities
    }
  }
}

Registration:

builder.Services.AddSingleton<IMetricRecordingFilter, OptionsBasedMetricRecordingFilter>();

How it works: 1. When an activity stops, the filter checks its Source.Name and OperationName 2. Matches against configured patterns using wildcards (*) 3. Returns true (record) or false (skip) based on first match

HttpHeadersSpanDurationMetricRecordingFilter

Enables dynamic filtering via HTTP headers - perfect for production debugging!

Use case: Enable metrics for specific requests without redeploying:

# Send request with header to enable metrics for this request
curl -H "Activity-Span-Recording: true" https://myapi.com/api/orders/123

How it works:

public class HttpHeadersSpanDurationMetricRecordingFilter : IMetricRecordingFilter
{
    public const string HeaderName = "Activity-Span-Recording";
    
    public virtual bool? ShouldRecord(Activity activity, Instrument instrument)
    {
        // Check if current HTTP request has the special header
        var httpContext = httpContextAccessor.HttpContext;
        if (httpContext?.Request.Headers.TryGetValue(HeaderName, out var value) == true)
        {
            return bool.TryParse(value, out var result) && result;
        }
        return null; // Defer to other filters
    }
}

Registration:

builder.Services.AddHttpContextAccessor();
builder.Services.AddSingleton<IMetricRecordingFilter, HttpHeadersSpanDurationMetricRecordingFilter>();

Benefits: - ✅ On-demand metrics for troubleshooting specific requests - ✅ No deployment required - toggle via HTTP headers - ✅ Safe for production - only affects requests with the header - ✅ Fine-grained control - per-request basis

Enriching Metrics with IMetricRecordingEnricher

Enrichers add contextual tags to metrics automatically.

The IMetricRecordingEnricher Interface

public interface IMetricRecordingEnricher
{
    Tags ExtractTags(Activity activity, Instrument instrument);
}

Purpose: Extract business-relevant tags from activities to enrich metrics.

OptionsBasedMetricRecordingEnricher

Configures which activity tags should become metric tags.

Configuration:

{
  "OptionsBasedMetricRecordingEnricher": {
    "MetricTags": [
      "customer_tier",
      "region",
      "deployment_environment",
      "service_version"
    ]
  }
}

How it works:

public virtual Tags ExtractTags(Activity activity, Instrument instrument)
{
    // Gets configured tag names from options
    var tagNames = options.Value.MetricTags;
    
    // Searches activity hierarchy for matching tags
    return tagNames
        .Select(tagName => {
            // Look in current activity and parent activities
            var value = activity.GetAncestors(includeSelf: true)
                .Select(a => a.GetTagItem(tagName))
                .FirstOrDefault(v => v != null);
            
            return new Tag(tagName, value);
        })
        .Where(tag => tag.Value != null);
}

Example usage:

using var activity = ActivitySource.StartRichActivity("ProcessOrder", new
{
    customer_tier = "premium",  // Will become metric tag
    region = "us-east",         // Will become metric tag
    order_id = "12345"          // Not in config, won't be included
});

// Resulting metric:
// diginsight.span_duration{
//   span_name="ProcessOrder",
//   customer_tier="premium",
//   region="us-east",
//   status="Ok"
// } = 150ms

Registration:

builder.Services.AddSingleton<IMetricRecordingEnricher, OptionsBasedMetricRecordingEnricher>();

Custom Enricher Example

Create custom enrichers for advanced scenarios:

public class BusinessContextEnricher : IMetricRecordingEnricher
{
    public Tags ExtractTags(Activity activity, Instrument instrument)
    {
        var tags = new List<Tag>();
        
        // Add deployment context
        tags.Add(new Tag("environment", Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")));
        
        // Add version information
        tags.Add(new Tag("version", Assembly.GetExecutingAssembly().GetName().Version?.ToString()));
        
        // Bucket high-cardinality values
        if (activity.GetTagItem("order_value") is double value)
        {
            var bucket = value switch
            {
                < 50 => "small",
                < 200 => "medium",
                < 1000 => "large",
                _ => "enterprise"
            };
            tags.Add(new Tag("order_value_bucket", bucket));
        }
        
        return tags;
    }
}

Custom Metric Recorders

Beyond span duration, you can create custom MetricRecorders for specialized metrics.

Example: HTTP Payload Size Recorder

public class HttpPayloadSizeRecorder : IActivityListenerLogic
{
    private readonly Histogram<long> requestSizeHistogram;
    private readonly Histogram<long> responseSizeHistogram;
    
    public HttpPayloadSizeRecorder(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("MyApp.Http");
        requestSizeHistogram = meter.CreateHistogram<long>("http.request.size", "bytes");
        responseSizeHistogram = meter.CreateHistogram<long>("http.response.size", "bytes");
    }
    
    public void ActivityStopped(Activity activity)
    {
        if (activity.Source.Name != "Microsoft.AspNetCore") return;
        
        var requestSize = activity.GetTagItem("http.request.body.size") as long? ?? 0;
        var responseSize = activity.GetTagItem("http.response.body.size") as long? ?? 0;
        
        var tags = new[]
        {
            new KeyValuePair<string, object?>("http.route", activity.GetTagItem("http.route")),
            new KeyValuePair<string, object?>("http.status_code", activity.GetTagItem("http.status_code"))
        };
        
        requestSizeHistogram.Record(requestSize, tags);
        responseSizeHistogram.Record(responseSize, tags);
    }
}

Example: Database Query Cost Recorder

public class DatabaseCostRecorder : IActivityListenerLogic
{
    private readonly Histogram<double> queryCostHistogram;
    
    public DatabaseCostRecorder(IMeterFactory meterFactory)
    {
        var meter = meterFactory.Create("MyApp.Database");
        queryCostHistogram = meter.CreateHistogram<double>("database.query.cost", "RU");
    }
    
    public void ActivityStopped(Activity activity)
    {
        if (!activity.Source.Name.StartsWith("Azure.Cosmos")) return;
        
        // Extract Request Units (RU) from Cosmos DB activity
        if (activity.GetTagItem("db.cosmosdb.request_charge") is double requestCharge)
        {
            var tags = new[]
            {
                new KeyValuePair<string, object?>("db.operation", activity.GetTagItem("db.operation")),
                new KeyValuePair<string, object?>("db.name", activity.GetTagItem("db.name"))
            };
            
            queryCostHistogram.Record(requestCharge, tags);
        }
    }
}

Registration:

services.AddSingleton<IActivityListenerLogic, HttpPayloadSizeRecorder>();
services.AddSingleton<IActivityListenerLogic, DatabaseCostRecorder>();

Complete Configuration Example

appsettings.json:

{
  "Diginsight": {
    "Activities": {
      "RecordSpanDuration": true,
      "SpanDurationMeterName": "MyApp.Telemetry",
      "SpanDurationMetricName": "operation.duration",
      "ActivitySources": {
        "MyApp.*": true,
        "Microsoft.EntityFrameworkCore": true,
        "Microsoft.AspNetCore": false
      }
    }
  },
  "OptionsBasedMetricRecordingFilter": {
    "ActivityNames": {
      "MyApp.Orders.*": true,
      "MyApp.Payment.*": true,
      "MyApp.Inventory.CheckAvailability": true,
      "System.*": false
    }
  },
  "OptionsBasedMetricRecordingEnricher": {
    "MetricTags": [
      "customer_tier",
      "region",
      "deployment_environment",
      "feature_flag"
    ]
  },
  "OpenTelemetry": {
    "Metrics": {
      "ExportIntervalMilliseconds": 5000,
      "MaxExportBatchSize": 512
    }
  }
}

Program.cs:

var builder = WebApplication.CreateBuilder(args);

// Add Diginsight with metrics
builder.Services.AddDiginsightCore();
builder.Services.AddSpanDurationMetricRecorder();

// Add filters and enrichers
builder.Services.AddSingleton<IMetricRecordingFilter, OptionsBasedMetricRecordingFilter>();
builder.Services.AddSingleton<IMetricRecordingFilter, HttpHeadersSpanDurationMetricRecordingFilter>();
builder.Services.AddSingleton<IMetricRecordingEnricher, OptionsBasedMetricRecordingEnricher>();

// Add custom metric recorders
builder.Services.AddSingleton<IActivityListenerLogic, HttpPayloadSizeRecorder>();
builder.Services.AddSingleton<IActivityListenerLogic, DatabaseCostRecorder>();

// Configure OpenTelemetry
builder.Services.AddOpenTelemetry()
    .WithMetrics(metrics => metrics
        .AddMeter("MyApp.*")
        .AddMeter("Diginsight.Diagnostics")
        .AddRuntimeInstrumentation()
        .AddHttpClientInstrumentation()
        .AddAspNetCoreInstrumentation()
        .AddPrometheusExporter()
        .AddApplicationInsightsExporter())
    .WithTracing(tracing => tracing
        .AddSource("MyApp.*")
        .AddSource("Diginsight.Diagnostics")
        .AddHttpClientInstrumentation()
        .AddAspNetCoreInstrumentation()
        .AddApplicationInsightsExporter());

var app = builder.Build();

// Enable Prometheus scraping endpoint
app.UseOpenTelemetryPrometheusScrapingEndpoint();

app.Run();

Application Code:

public class OrderService
{
    private static readonly ActivitySource ActivitySource = new("MyApp.Orders");
    
    public async Task<Order> ProcessOrderAsync(CreateOrderRequest request)
    {
        using var activity = ActivitySource.StartRichActivity("ProcessOrder", new
        {
            customer_id = request.CustomerId,
            customer_tier = request.CustomerTier,  // Will be enriched as tag
            region = request.Region,               // Will be enriched as tag
            item_count = request.Items.Count
        });
        
        try
        {
            var order = await ValidateAndCreateOrder(request);
            activity?.SetOutput(new { order.Id, order.Status });
            return order;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            throw;
        }
    }
}

Resulting metrics:

# Span duration with enriched tags
operation.duration{
  span_name="ProcessOrder",
  customer_tier="premium",
  region="us-east",
  status="Ok"
} = 250ms

# HTTP payload sizes
http.request.size{http.route="/api/orders", http.status_code="200"} = 1024 bytes
http.response.size{http.route="/api/orders", http.status_code="200"} = 4096 bytes

# Database costs
database.query.cost{db.operation="Query", db.name="OrdersDB"} = 12.5 RU

References

Official Documentation

• OpenTelemetry Metrics Specification
  The official OpenTelemetry metrics specification. Essential for understanding metric types (Counter, Histogram, Gauge), semantic conventions, and best practices for metric instrumentation.

• .NET Metrics API
  Microsoft’s documentation on System.Diagnostics.Metrics namespace. Covers Meter, Counter, Histogram creation and how Diginsight integrates with the native .NET metrics system.

• OpenTelemetry .NET SDK
  Official OpenTelemetry .NET implementation. Shows how to configure MeterProviders, exporters (Prometheus, OTLP, Application Insights), and metric aggregation.

Monitoring Backends

• Azure Application Insights Metrics
  Guide to querying and visualizing metrics in Application Insights. Explains how Diginsight metrics appear in Azure Monitor and how to create dashboards.

• Prometheus Querying
  Prometheus query language (PromQL) basics. Essential for creating alerts and dashboards from Diginsight metrics exported to Prometheus.

Best Practices

• High Cardinality Metrics Problem
  Excellent explanation of why high cardinality tags cause storage and performance issues. Critical reading for understanding why tag design matters in production.

• OpenTelemetry Semantic Conventions
  Standard attribute naming conventions for metrics. Following these conventions ensures consistency and interoperability when metrics are sent to various backends.

• Metric Naming Best Practices
  Industry standard for metric naming patterns. Helps design clear, consistent metric names that work well across different monitoring systems.

Diginsight Resources

• Diginsight GitHub Repository
  Official Diginsight repository with source code, samples, and documentation. Contains working examples of metric recorders, filters, and enrichers.

• Diginsight Samples
  Real-world sample applications demonstrating metric configuration, custom recorders, and integration with various backends.

See Also

• SpanDurationMetricRecorder Class
• OpenTelemetry Metrics Specification